Optimization of anemia treatment in hemodialysis patients via reinforcement learning

摘要

Objective: Anemia is a frequent comorbidity in hemodialysis patients that canbe successfully treated by administering erythropoiesis-stimulating agents(ESAs). ESAs dosing is currently based on clinical protocols that often do notaccount for the high inter- and intra-individual variability in the patient'sresponse. As a result, the hemoglobin level of some patients oscillates aroundthe target range, which is associated with multiple risks and side-effects.This work proposes a methodology based on reinforcement learning (RL) tooptimize ESA therapy. Methods: RL is a data-driven approach for solving sequential decision-makingproblems that are formulated as Markov decision processes (MDPs). Computingoptimal drug administration strategies for chronic diseases is a sequentialdecision-making problem in which the goal is to find the best sequence of drugdoses. MDPs are particularly suitable for modeling these problems due to theirability to capture the uncertainty associated with the outcome of the treatmentand the stochastic nature of the underlying process. The RL algorithm employedin the proposed methodology is fitted Q iteration, which stands out for itsability to make an efficient use of data. Results: The experiments reported here are based on a computational modelthat describes the effect of ESAs on the hemoglobin level. The performance ofthe proposed method is evaluated and compared with the well-known Q-learningalgorithm and with a standard protocol. Simulation results show that theperformance of Q-learning is substantially lower than FQI and the protocol. Conclusion: Although prospective validation is required, promising resultsdemonstrate the potential of RL to become an alternative to current protocols.